{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "5120ef0f-8510-4ad7-a0ea-2e306aec0064",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Transferring Local Files During Workflows\n",
    "\n",
    "Transfer files locally before or after executing an electron.\n",
    "\n",
    "### Prerequisites\n",
    "\n",
    "1. For convenience in running this example, define the read (source) and write (destination) file paths. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "5a8139da",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "# Define source and destination filepaths \n",
    "source_filepath = Path('./my_source_file').resolve()\n",
    "dest_filepath = Path('./my_dest_file').resolve()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a34a6156",
   "metadata": {},
   "source": [
    "2. Create a source file to transfer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "089aa573",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create an example file\n",
    "file_content = \"\"\"Mares eat oats and does eat oats\n",
    "And little lambs eat ivy ...\n",
    "\"\"\"\n",
    "with open(source_filepath, \"w\") as f:\n",
    "    f.write(file_content)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26a649dd",
   "metadata": {},
   "source": [
    "### Procedure\n",
    "\n",
    "1. Define a Covalent `FileTransfer` object, assigning the source and destination file paths respectively as its arguments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "34e40034",
   "metadata": {},
   "outputs": [],
   "source": [
    "import covalent as ct\n",
    "\n",
    "xfer = ct.fs.FileTransfer(str(source_filepath), str(dest_filepath))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c8e8a7c",
   "metadata": {},
   "source": [
    "2. Define a list of Covalent `FileTransfer` objects to assign to a task. (In this example, the list contains only the single `FileTransfer` named `xfer`.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "279dc3fa",
   "metadata": {},
   "outputs": [],
   "source": [
    "ft_list = [xfer]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf8e61e2",
   "metadata": {},
   "source": [
    "3. Define an electron that uses a Covalent `FileTransfer` task to read the source file and writes to the destination file, assigning the list containing the `FileTransfer` objects to the `files` argument of the electron decorator. (Note that the `files` argument takes a list of Covalent `FileTransfer` objects, *not* files or path names.) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "de6ed992",
   "metadata": {},
   "outputs": [],
   "source": [
    "@ct.electron(\n",
    "        files = ft_list\n",
    ")\n",
    "def my_file_transfer_task(files):\n",
    "    from_file, to_file = files[0]\n",
    "    with open(to_file,'w') as f:\n",
    "        for line in open(from_file, 'r'):\n",
    "            f.write(line)\n",
    "    return to_file"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0764f4a9",
   "metadata": {},
   "source": [
    "Here is the task definition again, with the three steps combined in the electron decorator. The `FileTransfer` defaults to the local `Rsync` strategy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "789a74ef",
   "metadata": {},
   "outputs": [],
   "source": [
    "import covalent as ct\n",
    "\n",
    "@ct.electron(\n",
    "        files=[ct.fs.FileTransfer(str(source_filepath), str(dest_filepath))] # defaults to Rsync\n",
    ")\n",
    "def my_file_transfer_task(files=None):\n",
    "    from_file, to_file = files[0]\n",
    "    with open(to_file,'w') as f_to, open(from_file, 'r') as f_from:\n",
    "        for line in f_from:\n",
    "            f_to.write(line)\n",
    "    return to_file"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1616c355",
   "metadata": {},
   "source": [
    "4. Run the electron thus created in a lattice:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "a30852d1-2fa4-4ffb-b01d-7e7cc9925182",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create and dispatch a workflow to transfer data from source to destination, and write to destination file\n",
    "\n",
    "@ct.lattice()\n",
    "def my_workflow():\n",
    "    return my_file_transfer_task()\n",
    "\n",
    "dispatch_id = ct.dispatch(my_workflow)()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79b3012b-ecf8-4c96-a0c7-a3b11d69a10e",
   "metadata": {},
   "source": [
    "5. Confirm the transfer by reading the contents of the destination file:\n",
    "After executing the workflow a copy of the file (`source_filepath`) has been written to `my_dest_file`. This file transfer occurred before electron execution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "0a6a94bf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Lattice Result\n",
      "==============\n",
      "status: COMPLETED\n",
      "result: /Users/sankalpsanand/dev/covalent/doc/source/how_to/coding/my_dest_file\n",
      "input args: \n",
      "input kwargs: \n",
      "error: None\n",
      "\n",
      "start_time: 2024-01-05 14:48:31.791928\n",
      "end_time: 2024-01-05 14:48:32.501517\n",
      "\n",
      "results_dir: /Users/sankalpsanand/.cache/covalent/results/bbd6f2b0-ecb4-4e05-ba63-6c10da81ceeb\n",
      "dispatch_id: bbd6f2b0-ecb4-4e05-ba63-6c10da81ceeb\n",
      "\n",
      "Node Outputs\n",
      "------------\n",
      "my_file_transfer_task(0): /Users/sankalpsanand/dev/covalent/doc/source/how_to/coding/my_dest_file\n",
      ":postprocess:reconstruct(1): /Users/sankalpsanand/dev/covalent/doc/source/how_to/coding/my_dest_file\n",
      "\n",
      "Reading from  /Users/sankalpsanand/dev/covalent/doc/source/how_to/coding/my_dest_file \n",
      "\n",
      "Mares eat oats and does eat oats\n",
      "And little lambs eat ivy ...\n",
      "\n"
     ]
    }
   ],
   "source": [
    "result = ct.get_result(dispatch_id, wait=True)\n",
    "print(result)\n",
    "result_filepath = result.result\n",
    "\n",
    "# Read from the destination file\n",
    "print(\"Reading from \", result_filepath, \"\\n\")\n",
    "with open(result_filepath,'r') as f:\n",
    "    print(f.read())\n",
    "\n",
    "# Clean up files\n",
    "source_filepath.unlink()\n",
    "dest_filepath.unlink()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8fec2999",
   "metadata": {},
   "source": [
    "### See Also\n",
    "\n",
    "[Transferring Remote Files During Workflows](./file_transfers_to_from_remote.ipynb)\n",
    "\n",
    "[Transferring Files to and from an S3 Bucket](./file_transfers_to_from_s3.ipynb)\n",
    "\n",
    "[Transferring Files To and From Azure Blob Storage](./file_transfers_to_from_azure_blob.ipynb)\n",
    "\n",
    "[Transferring Files To and From Google Cloud Storage](./file_transfers_to_from_gcp_storage.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.18"
  },
  "vscode": {
   "interpreter": {
    "hash": ""
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
